{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1GhdrwnPhxpM"
   },
   "source": [
    "# Regression on Gradient Boosting: CPU vs GPU\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/catboost/tutorials/blob/master/tools/google_colaboratory_cpu_vs_gpu_regression_tutorial.ipynb)\n",
    "\n",
    "This is a basic tuturoal which shows how to run regression on gradient boosting on CPU and GPU on Google Colaboratory. It will give you an opportunity to see the speedup that you get from GPU training. The speedup is large even on Tesla K80 that is available in Colaboratory. On newer generations of GPU the speedup will be much bigger.\n",
    "\n",
    "We will use CatBoost gradient boosting library, which is known for it's good GPU performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cSWEX-NYh7-U"
   },
   "source": [
    "# !Set GPU as hardware accelerator!\n",
    "First of all, you need to select GPU as hardware accelerator. There are two simple steps to do so:  \n",
    "#### Step 1. Navigate to 'Runtime' menu and select 'Change runtime type'  \n",
    "#### Step 2. Choose GPU as hardware accelerator.  \n",
    "That's all!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s4EMTHRqiDE1"
   },
   "source": [
    "## Importing CatBoost\n",
    "\n",
    "Next big thing is to import CatBoost inside environment. Colaboratory has built in libraries installed and most libraries can be installed quickly with a simple  `!pip install` command.  \n",
    "Please take notice you need to re-import library every time you starts new session of Colab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "colab_type": "code",
    "id": "jeslyTF6peOo",
    "outputId": "8a751506-2512-4d23-95da-0c87cf483d36"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: catboost in /usr/local/lib/python3.6/dist-packages (0.12.2)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from catboost) (1.11.0)\n",
      "Requirement already satisfied: enum34 in /usr/local/lib/python3.6/dist-packages (from catboost) (1.1.6)\n",
      "Requirement already satisfied: pandas>=0.19.1 in /usr/local/lib/python3.6/dist-packages (from catboost) (0.22.0)\n",
      "Requirement already satisfied: numpy>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from catboost) (1.14.6)\n",
      "Requirement already satisfied: python-dateutil>=2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.19.1->catboost) (2.5.3)\n",
      "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.19.1->catboost) (2018.9)\n"
     ]
    }
   ],
   "source": [
    "!pip install catboost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_1rchBYFm4Je"
   },
   "source": [
    "## Including libraries\n",
    "Now we need to include the libraries:\n",
    "\n",
    "`CatBoostRegressor` - for regression,\n",
    "\n",
    "`timeit` - to measure time,\n",
    "\n",
    "`make_regression` - to generate dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4BzUM4I8qBgR"
   },
   "outputs": [],
   "source": [
    "from catboost import CatBoostRegressor\n",
    "import timeit\n",
    "from sklearn.datasets import make_regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GPx1En1liM3e"
   },
   "source": [
    "## Generating dataset\n",
    "The next step is dataset generating. GPU training is useful for large datsets. You will get a good speedup starting from 10k objects.\n",
    "Because of that reason we have generated a large dataset  (40.000 documents and 2.000 features) for this tutorial.\n",
    "\n",
    "We will generate a dataset using the  `datasets.make_regression` module from the `sklearn` library, because this is the easiest way to load a large dataset into Google Kolab for our tests. The dataset is a linear regression with Gaussian noise.\n",
    "\n",
    "The code below does this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iY4R_dRnqIcs"
   },
   "outputs": [],
   "source": [
    "num_rows = 40000\n",
    "num_colomns = 2000\n",
    "X_train, y_train = make_regression(n_samples=num_rows, n_features=num_colomns, \n",
    "                                   bias=100, noise=1.0, random_state=0)  \n",
    "X_test, y_test = X_train, y_train "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BudTE2Pmik6i"
   },
   "source": [
    "## Training on CPU\n",
    "Now we will train the model on CPU and measure execution time.\n",
    "We will use 100 iterations for our CPU training since otherwise it will take a long time.\n",
    "It will take around 8 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 303
    },
    "colab_type": "code",
    "id": "G5r097DivJ0S",
    "outputId": "0614ae36-bf37-41c9-8173-cf65fe38897a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 210.7406779\ttest: 210.7406779\tbest: 210.7406779 (0)\ttotal: 5.42s\tremaining: 8m 56s\n",
      "10:\tlearn: 175.7370024\ttest: 175.7370024\tbest: 175.7370024 (10)\ttotal: 51.3s\tremaining: 6m 55s\n",
      "20:\tlearn: 149.6425348\ttest: 149.6425348\tbest: 149.6425348 (20)\ttotal: 1m 34s\tremaining: 5m 56s\n",
      "30:\tlearn: 130.0950162\ttest: 130.0950162\tbest: 130.0950162 (30)\ttotal: 2m 18s\tremaining: 5m 8s\n",
      "40:\tlearn: 114.8508041\ttest: 114.8508041\tbest: 114.8508041 (40)\ttotal: 3m\tremaining: 4m 19s\n",
      "50:\tlearn: 102.8672930\ttest: 102.8672930\tbest: 102.8672930 (50)\ttotal: 3m 41s\tremaining: 3m 33s\n",
      "60:\tlearn: 93.1019234\ttest: 93.1019234\tbest: 93.1019234 (60)\ttotal: 4m 23s\tremaining: 2m 48s\n",
      "70:\tlearn: 84.9256561\ttest: 84.9256561\tbest: 84.9256561 (70)\ttotal: 5m 5s\tremaining: 2m 4s\n",
      "80:\tlearn: 77.9872987\ttest: 77.9872987\tbest: 77.9872987 (80)\ttotal: 5m 46s\tremaining: 1m 21s\n",
      "90:\tlearn: 72.1042340\ttest: 72.1042340\tbest: 72.1042340 (90)\ttotal: 6m 26s\tremaining: 38.3s\n",
      "99:\tlearn: 67.4124799\ttest: 67.4124799\tbest: 67.4124799 (99)\ttotal: 7m 3s\tremaining: 0us\n",
      "\n",
      "bestTest = 67.41247986\n",
      "bestIteration = 99\n",
      "\n",
      "Time to fit model on CPU: 464 sec\n"
     ]
    }
   ],
   "source": [
    "def train_on_cpu():  \n",
    "  model = CatBoostRegressor(\n",
    "    iterations=100,\n",
    "    learning_rate=0.03\n",
    "  )\n",
    "  \n",
    "  model.fit(\n",
    "      X_train, y_train,\n",
    "      eval_set=(X_test, y_test),\n",
    "      verbose=10\n",
    "  );   \n",
    "      \n",
    "cpu_time = timeit.timeit('train_on_cpu()', \n",
    "                         setup=\"from __main__ import train_on_cpu\", \n",
    "                         number=1)\n",
    "\n",
    "print('Time to fit model on CPU: {} sec'.format(int(cpu_time)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YFJTWgfZiygP"
   },
   "source": [
    "## Training on GPU\n",
    "The previous code execution has been done on CPU. It's time to use GPU!  \n",
    "We need to use '*task_type='GPU'*' parameter value to run GPU training. Now the execution time wouldn't be so big :)  \n",
    "BTW if Colaboratory shows you a warning 'GPU memory usage is close to the limit', just press 'Ignore'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 267
    },
    "colab_type": "code",
    "id": "uhS9Wb59kClq",
    "outputId": "84cd57a2-e1db-4720-9f86-1466781ec9fc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 210.6864742\ttest: 210.6864438\tbest: 210.6864438 (0)\ttotal: 418ms\tremaining: 41.4s\n",
      "10:\tlearn: 175.3974321\ttest: 175.3974048\tbest: 175.3974048 (10)\ttotal: 3.75s\tremaining: 30.3s\n",
      "20:\tlearn: 149.2556840\ttest: 149.2556786\tbest: 149.2556786 (20)\ttotal: 6.84s\tremaining: 25.7s\n",
      "30:\tlearn: 129.5295858\ttest: 129.5295858\tbest: 129.5295858 (30)\ttotal: 9.87s\tremaining: 22s\n",
      "40:\tlearn: 114.3826525\ttest: 114.3826420\tbest: 114.3826420 (40)\ttotal: 12.9s\tremaining: 18.5s\n",
      "50:\tlearn: 102.3210672\ttest: 102.3210711\tbest: 102.3210711 (50)\ttotal: 15.8s\tremaining: 15.2s\n",
      "60:\tlearn: 92.4963048\ttest: 92.4963048\tbest: 92.4963048 (60)\ttotal: 18.8s\tremaining: 12s\n",
      "70:\tlearn: 84.2915749\ttest: 84.2915654\tbest: 84.2915654 (70)\ttotal: 21.7s\tremaining: 8.86s\n",
      "80:\tlearn: 77.3639503\ttest: 77.3639528\tbest: 77.3639528 (80)\ttotal: 24.5s\tremaining: 5.75s\n",
      "90:\tlearn: 71.4349886\ttest: 71.4349942\tbest: 71.4349942 (90)\ttotal: 27.3s\tremaining: 2.7s\n",
      "99:\tlearn: 66.7548260\ttest: 66.7548230\tbest: 66.7548230 (99)\ttotal: 29.8s\tremaining: 0us\n",
      "bestTest = 66.75482305\n",
      "bestIteration = 99\n",
      "Time to fit model on GPU: 69 sec\n"
     ]
    }
   ],
   "source": [
    "def train_on_gpu():  \n",
    "  model = CatBoostRegressor(\n",
    "    iterations=100,\n",
    "    learning_rate=0.03,\n",
    "    task_type='GPU'\n",
    "  )\n",
    "  model.fit(\n",
    "      X_train, y_train,\n",
    "      eval_set=(X_test, y_test),\n",
    "      verbose=10\n",
    "  );      \n",
    "      \n",
    "gpu_time = timeit.timeit('train_on_gpu()', \n",
    "                         setup=\"from __main__ import train_on_gpu\", \n",
    "                         number=1)\n",
    "\n",
    "print('Time to fit model on GPU: {} sec'.format(int(gpu_time)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "wiilD1uZlSmp",
    "outputId": "5cc6cf87-bd8e-42df-c7a2-51aeeceb3f98"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU speedup over CPU: 6.71x\n"
     ]
    }
   ],
   "source": [
    "print('GPU speedup over CPU: ' + '%.2f' % (cpu_time/gpu_time) + 'x')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gw7qmALMi_yi"
   },
   "source": [
    "As you can see GPU is much faster than CPU on large datasets. It takes just 1 - 2 mins vs 7 - 8 mins to fit the model.\n",
    "This is a good reason to use GPU instead of CPU!\n",
    "  \n",
    "Thank you for attention! "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "google_colaboratory_cpu_vs_gpu_regression_tutorial.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
